%md-sandbox

<div style="text-align: center; line-height: 0; padding-top: 9px;">

  <img src="https://databricks.com/wp-content/uploads/2018/03/db-academy-rgb-1200px.png" alt="Databricks Learning" style="width: 1200px">

</div>

%md

# Data Exploration

In this notebook, we will use the dataset we cleansed in the previous lab to do some Exploratory Data Analysis (EDA).

This will help us better understand our data to make a better model.

## ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) In this lesson you:<br>

 - Identify log-normal distributions

 - Build a baseline model and evaluate

%md

Let's keep 80% for the training set and set aside 20% of our data for the test set. We will use the `randomSplit` method [Python](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.randomSplit)/[Scala](https://spark.apache.org/docs/latest/api/scala/#org.apache.spark.sql.Dataset).

We will discuss more about the train-test split later, but throughout this notebook, do your data exploration on `trainDF`.

%md

Let's make a histogram of the price column to explore it (change the number of bins to 300).  

%md

Is this a <a href="https://en.wikipedia.org/wiki/Log-normal_distribution" target="_blank">Log Normal</a> distribution? Take the `log` of price and check the histogram. Keep this in mind for later :).

%md

Now take a look at how `price` depends on some of the variables:

* Plot `price` vs `bedrooms`

* Plot `price` vs `accommodates`

Make sure to change the aggregation to `AVG`.

%md

Let's take a look at the distribution of some of our categorical features

%md

Which neighbourhoods have the highest number of rentals? Display the neighbourhoods and their associated count in descending order.